Dental image processing protocol for dental aligners

ABSTRACT

The present disclosure relates to a method of pixel-based classification of medical images. The method includes training a neural network to perform the pixel-based classification, the training comprising performing a first training on a classifier using a first training set of medical images, pixels of the first training set of medical images being manually labeled as a biological structure type, applying the classifier after the first training to a second training set of medical, identifying and correcting incorrectly classified pixels of the second training set of medical images, and performing a second training on the classifier after the first training using the manually labeled first training set and the correctly classified second training set, and applying the classifier after the second training to the medical images to perform the pixel-based classification.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No.16/929,588, filed Jul. 15, 2020, which is a continuation of U.S.application Ser. No. 16/017,687, filed Jun. 25, 2018, (now U.S. Pat. No.10,748,651), which claims priority to Eurasian Patent Application No.201700561, filed Nov. 16, 2017, the content of which is herebyincorporated herein by reference in its entirety.

BACKGROUND Field of the Disclosure

The present disclosure relates to a dental image processing protocol forthe design of dental aligners.

Description of the Related Art

Orthodontics, generally, and dental alignment, in particular, is awell-developed area of dental care. For patients with maligned teeth,traditional braces or, more recently, clear aligners, offer a strategyfor improved dental function and aesthetics through gradual teethmovements. These gradual teeth movements slowly move a crown of a toothuntil a desired final position is reached. These approaches, however,fail to appropriately consider the impact of corresponding rootmovements, in the context of surrounding soft and hard tissues, on thefinal position of the crown, focusing instead on an aesthetically andfunctionally ideal crown position. An approach for determining crownposition that adequately incorporates the impact of root movement andthe root environment has yet to be developed.

The foregoing “Background” description is for the purpose of generallypresenting the context of the disclosure. Work of the inventors, to theextent it is described in this background section, as well as aspects ofthe description which may not otherwise qualify as prior art at the timeof filing, are neither expressly or impliedly admitted as prior artagainst the present invention.

SUMMARY

The present disclosure relates to a method, apparatus, andcomputer-readable medium comprising a processing circuitry configured toclassify pixels of one or more medical images into classes correspondingto biological structure types, segment the classified pixels of the oneor more medical images into biological structures, render athree-dimensional model of the biological structures based on thesegmented classified pixels, determine one or more metrics, based uponthe three-dimensional model, describing a bone of the biologicalstructures, acquire a final position of each of the one or more teeth ofthe dental arch based upon the three-dimensional model, and generate theintermediate position of each of the one or more teeth of the patientbased upon the one or more metrics and the acquired final position.

According to an embodiment, the present disclosure further relates to amethod of pixel-based classification of medical images, comprisingtraining a neural network to perform the pixel-based classification ofthe medical images, the training comprising performing a first trainingon a classifier using a first training set of medical images, pixels ofthe first training set of medical images being manually labeled,applying the classifier after the first training to a second trainingset of medical images to classify pixels of the second training set ofmedical images, identifying and correcting incorrectly classified pixelsof the second training set of medical images, and performing a secondtraining on the classifier after the first training using the manuallylabeled first training set of medical images and the correctlyclassified second training set of medical images; and applying theclassifier after the second training to the medical images to performthe pixel-based classification, the pixel-based classification of themedical images including assigning pixels of each medical image to abiological structure type.

The foregoing paragraphs have been provided by way of generalintroduction, and are not intended to limit the scope of the followingclaims. The described embodiments, together with further advantages,will be best understood by reference to the following detaileddescription taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the office upon request and paymentof the necessary fee.

A more complete appreciation of the disclosure and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 is an illustration of dental aligners, according to an embodimentof the present disclosure;

FIG. 2 is an illustration of a dental image of a tooth, according to anembodiment of the present disclosure;

FIG. 3 is a flowchart of a dental image processing protocol, accordingto an embodiment of the present disclosure;

FIG. 4 is an illustration of a complex three-dimensional model generatedfrom a plurality of processed dental images and annotated with a surfaceheat map, according to an embodiment of the present disclosure;

FIG. 5 is a flowchart of an aspect of a training protocol of a dentalimage processing protocol, according to an embodiment of the presentdisclosure;

FIG. 6A is an illustration of an aspect of a manually-segmented dentalimage of a tooth, according to an embodiment of the present disclosure;

FIG. 6B is an illustration of an aspect of a manually-segmented dentalimage of a tooth, according to an embodiment of the present disclosure;

FIG. 7A is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7B is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7C is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7D is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7E is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7F is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7G is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7H is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7I is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7J is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7K is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7L is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7M is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7N is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7O is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7P is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7Q is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7R is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7S is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7T is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 7U is an illustration of an exemplary pixel patch of a dental imageprocessing protocol, according to an embodiment of the presentdisclosure;

FIG. 8A is an illustration of an aspect of a snakes segmentation of adental image processing protocol, according to an embodiment of thepresent disclosure;

FIG. 8B is an illustration of an aspect of a snakes segmentation of adental image processing protocol, according to an embodiment of thepresent disclosure;

FIG. 8C is an illustration of an aspect of a snakes segmentation of adental image processing protocol, according to an embodiment of thepresent disclosure;

FIG. 8D is an illustration of an aspect of a snakes segmentation of adental image processing protocol, according to an embodiment of thepresent disclosure;

FIG. 9 is a flowchart of an aspect of a training data generationprotocol, according to an embodiment of the present disclosure;

FIG. 10A is an illustration of a segmentation of teeth of a dentalimage, according to an embodiment of the present disclosure;

FIG. 10B is an illustration of a segmentation of teeth of a dentalimage, according to an embodiment of the present disclosure;

FIG. 10C is an illustration of a segmentation of teeth of a dentalimage, according to an embodiment of the present disclosure;

FIG. 11A is an illustration of a source image of a segmentation of boneof a dental image, according to an embodiment of the present disclosure;

FIG. 11B is an illustration of a segmentation of bone of a dental image,according to an embodiment of the present disclosure;

FIG. 11C is an illustration of a segmentation of bone of a dental image,according to an embodiment of the present disclosure;

FIG. 12A is a flowchart of a determination of a distance metric of athree-dimensional model, according to an embodiment of the presentdisclosure;

FIG. 12B is a flowchart of a determination of a density metric of athree-dimensional model, according to an embodiment of the presentdisclosure;

FIG. 13A is an illustration of a three-dimensional model of dentition,according to an embodiment of the present disclosure;

FIG. 13B is an illustration of a three-dimensional model of dentition,according to an embodiment of the present disclosure;

FIG. 14A is an illustration of a three-dimensional model of an initialtooth position, according to an embodiment of the present disclosure;

FIG. 14B is an illustration of a three-dimensional model of anintermediary tooth position, according to an embodiment of the presentdisclosure;

FIG. 14C is an illustration of a three-dimensional model of a finaltooth position, according to an embodiment of the present disclosure;and

FIG. 15 is a schematic of exemplary hardware for implementation of adental image processing protocol, according to an embodiment of thepresent disclosure.

DETAILED DESCRIPTION

The terms “a” or “an”, as used herein, are defined as one or more thanone. The term “plurality”, as used herein, is defined as two or morethan two. The term “another”, as used herein, is defined as at least asecond or more. The terms “including” and/or “having”, as used herein,are defined as comprising (i.e., open language). Reference throughoutthis document to “one embodiment”, “certain embodiments”, “anembodiment”, “an implementation”, “an example” or similar terms meansthat a particular feature, structure, or characteristic described inconnection with the embodiment is included in at least one embodiment ofthe present disclosure. Thus, the appearances of such phrases or invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments without limitation.

Currently, orthodontist and dental technicians develop teeth movementplans based upon initial and ideal final crown positions. Consideringonly teeth crowns, however, ignoring root movements and the rootenvironment, makes it possible that root collisions or other damage mayoccur to the tooth or surrounding bone tissue at intermediary toothpositions. In an example, an intermediary tooth position may result incollision of the roots of adjacent teeth. In another example, athickness of the alveolar process, the bone tissue surrounding a toothroot, can limit the ability of the tooth root to move to an intermediarystep. When the surrounding bone tissue is of insufficient thickness, arealizable movement of a tooth may be less than an ideal movement of thetooth, resulting in impaired treatment and sub-optimal teeth functionand aesthetic. In yet another example, varying densities of periodontalbone can impact potential root movements and realignment, thereof.

Based upon the insufficiencies of current methodologies, the presentdisclosure describes an orthodontic treatment approach that considers anevaluation of the condition of the tissues surrounding the tooth and thealveolar process, in particular. Moreover, the evaluation of thecondition of the tissues surrounding the tooth is patient-specific,reflecting the unique density and thickness of an individual patient'speriodontal bone.

FIG. 1 is an illustration of dental aligners, according to an embodimentof the present disclosure. Following a determination of an initial toothposition and a realizable final tooth position, intermediary toothmovements can be determined and dental aligners 100 can be fabricated,accordingly, to slowly move and realign a patient's teeth. Oftentimes,however, as described above, these determinations are based only upon anideal final tooth position and the position of the crown of the toothrelative to adjacent teeth, which can lead to root, periodontalligament, or periodontal bone damage upon movement. In order toincorporate information related to the environment of the tooth andsurrounding tissues during dental aligner 100 fabrication, an approachfor identifying tissue-types, generating three-dimensional (3D) models,and determining periodontal tissue characteristics, thereof, isrequired. To this end, it becomes necessary to develop a strategy fordiscerning soft tissues from hard tissues and tooth roots fromsurrounding bone of varying densities. FIG. 2 is an illustration of adental image of a tooth, according to an embodiment of the presentdisclosure. In an embodiment, a dental image of a tooth may be but isnot limited to an image acquired via intraoral optical imaging,impressions, dental models, ultrasound, or radiography. In an example, aplurality of images, or slices, may be acquired via radiography andreconstructed to render a 3D model. With reference again to FIG. 2, atooth 204 comprises a crown 206 and one or more roots 205. The one ormore roots 205 are resident within an alveolar process, a thickenedridge of bone containing dental alveoli, or tooth sockets. The alveolarprocess is comprised of cortical bone 215, a compact, relatively densebone, and cancellous bone 210, a spongy, relatively porous bone.Together, cortical bone 215 and cancellous bone 210 provide a strongfoundation from which the one or more roots 205 of the tooth 204 areanchored. As related to the present disclosure, cortical bone 215 andcancellous bone 210, as periodontal tissues, contribute to thedetermination of possible movements of a tooth.

In planning a tooth movement such that the tooth and periodontalenvironment are considered concurrently, a variety of structures,including the above-described features, must be identified. Moreover,once these features have been identified for a single two-dimensional(2D) dental, image, the same can be performed for additional 2D dentalimages, or slices, until a 3D model can be rendered, therefrom. Inaddition to providing for aesthetic evaluation, a 3D model synthesizesinformation regarding periodontal tissue density and thickness, therebybounding possible tooth movements and providing a prescribing dentalprofessional a tool from which to determine a tooth movement. Theprocess alluded to above is described in FIG. 3, a flowchart of a dentalimage processing protocol implemented via a dental image processingdevice comprising a processing circuitry.

According to an embodiment of the present disclosure, the dental imageprocessing protocol described herein can be appreciated in context of afull dental arch or an individual tooth. Initially, digitalrepresentations of an initial position of a patient's teeth must beacquired. A variety of techniques including but not limited toimpressions, dental scanner of impressions or dental models, intraoralscanners for digital impressions, intraoral X-ray, ultrasound, andcomputed tomography can be used individually or in combination toacquire digital representations of the initial position of the patient'steeth. In an embodiment of the present disclosure, in order to create adigital impression, an intraoral scanner S341 may be employed to acquiretopographical characteristics of crowns of the teeth. The intraoralscanner may employ a modality selected from a group including but notlimited to lasers, infrared light, and structured light. So that toothmovements can be determined in the context of crowns and roots, aradiographic imaging modality may be employed in order to acquirespatial information relating to the roots and periodontal tissues,including soft tissues and hard tissues (e.g. alveolar process). In anembodiment, the radiographic imaging modality may be selected from thegroup including but not limited to projection radiography, computedtomography (CT), dual energy X-ray absorptiometry, fluoroscopy, andcontrast radiography. In an example, the radiographic imaging modalitymay be cone beam computed tomography (CBCT) S350. Radiographic imagesmay comprise multi-planar radiographic images including but not limitedto sagittal, transverse, and coronal. It should be appreciated that,apart from radiographic techniques, a variety of imaging modalitiesincluding but not limited to ultrasound may be used for acquisition ofimages describing spatial information of the roots and periodontaltissues.

Following acquisition of a plurality of dental images of a patient viaCBCT, various biological structures, including the teeth and the jaw,must be digitally identified so that they can be later incorporated intoa holistic 3D model of the dental environment. As an alternative tomanual classification of individual biological structures of a pluralityof dental images, the present disclosure employs a machine learningapproach, a platform for rapid evaluation of the plurality of dentalimages, to classify the various biological structures of each dentalimage of a patient S351. In an exemplary embodiment, the machinelearning approach may be a fully convolutional neural network (FCN).Unlike similar approaches that perform patch-wise predictions, an FCNclassifier evaluates and predicts the classification of each pixel of anunknown image. Per-pixel classification allows the resulting predictionsto be segmented into multiple tissue classes S352, combining adjacentpixels of similar classification and density and defining the shape ofeach type of tissue, or biological structure, as building blocks for a3D model. In turn, via a surface reconstruction technique such as, forinstance, marching cubes, each of the plurality of classified andsegmented dental images of a patient are combined and reconstructed toform a 3D model of dentition, including a patient's dental arches andsurrounding tissues S353. In an exemplary embodiment, the surfacereconstruction technique may be selected from a group including but notlimited to marching tetrahedrons and marching cubes, a sequentialsurface rendering model wherein a polygonal mesh is fitted to a surfaceinteracting with pixels from adjacent slices.

The reconstructed surfaces can then be integrated into the digitalimpressions acquired via intraoral 3D imaging to create a simple 3Dmodel of the mouth of a patient S343, including crowns, roots andperiodontal tissues. In an embodiment, this data integration may beaccomplished via point-based alignment of two surface models, aninteractive method of registration of polygonal meshes. First, at leastthree corresponding points on each of the two surface models areselected. Next, a transformation matrix is computed and applied viatranslation and rotation or quaternion. If the resulting overlap betweenthe two surface models is greater than a pre-determined threshold, a newtransformation matrix must be determined and applied such that theresulting overlap between the two surface models is less than thepre-determined threshold. In an exemplary embodiment, the at least threecorresponding points on each of the two surface models are selectedmanually. In another embodiment, corresponding points on each of the twosurface models may be selected automatically via software, wherein thecorresponding points are features selected from a group including butnot limited to cusps, gloves, offsets and pits of molars, or centralpoints of cutting edges on incisors.

Concurrently, in order to better define the periodontal environment ofthe simple 3D model and provide a realistic model of potential tooth androot movement, characteristics of the alveolar process must bedetermined. To this end, a density measurement S354 and a distancemeasurement S355 of the periodontal space may be computed from thesurface reconstruction, or simple 3D model, of the classified andsegmented predictions. The density measurement comprises computing, fromeach point on a mesh describing the surface of a root, a metric of thedensity of the surrounding tissue. This metric may be determined on thebasis of mean voxel intensities surrounding a vertex-point coordinate ofthe segmentation, reflecting the spatial qualities of bone and theability of a tooth to move, therein. The distance measurement comprisescomputing, for each point on the mesh describing the surface of a bone,such as the buccal surface of the alveolar bone, a distance to a nearestpoint on the mesh describing the root. This distance, therefore,reflects a volume of the alveolar process wherein a tooth may move.

Once calculated for each point within the mesh, or model, the densitymeasurement and distance measurement may be mapped to the simple 3Dmodel, creating a complex 3D model S356. To allow for rapidvisualization of the distance measurement, varying distances are denotedvia heat map, wherein regions of minimal thickness and regions ofmaximal thickness are represented with varying colors.

Having now generated a complex 3D model of a patient's mouth, includingdental arches and alveolar process, annotated to denote tissuethicknesses and densities, the complex 3D model may be manipulated todetermine realizable final tooth positions S360. In doing so,intermediary tooth positions and movements necessary to achieve suchpositions may be determined. FIG. 4 is an illustration of a complex 3Dmodel generated from a plurality of processed dental images, annotatedwith a surface heat map, according to an embodiment of the presentdisclosure.

As described above, following acquisition and processing of intraoral 3Dscans and radiographic images, surface mesh data may be integrated tocreate a 3D model of an initial position of the dental arches of apatient. A heat map, overlaid on the 3D model, indicates localthicknesses of the alveolar process, the periodontal environment thereinvarying across individual teeth of the dental arches. For example, withreference to FIG. 4, a canine 408 and an adjacent premolar 407 havedisparate local periodontal environments. The canine 408 may bepositioned closer to a buccal surface of an alveolar process 409, asindicated by a darker shade, intense red, while the premolar 407 may bepositioned posteriorly with respect to the buccal surface of thealveolar process 409, proximate to a lingual surface of the alveolarprocess 409, as indicated by light shades of the heat map. This heat mapfeature allows a prescribing dental professional to visualize possibleand impossible tooth movements and select appropriate intermediarymovements within skeletal constraints.

Critical to the success of the dental image processing protocol is theability to accurately classify biological structures, or tissues, ofdental images, and radiographic slices, in particular. To this end, FIG.5 is a flowchart of an aspect of a training protocol of a classificationapproach of a dental image processing protocol, according to anembodiment of the present disclosure. Generally, the training approachprepares an FCN classifier to be applied to the binary classification of‘bone’ or the binary classification of ‘teeth’. Specifically, thetraining approach provides annotated training data, manually orautomatically generated, directed to the above-described classes. In anexample, when applied, the FCN classifier is meant to predict, for eachpixel of a slice, class 1 if a tooth is present and class 0 if a toothis not present.

The process of generating the annotated training data is described inFIG. 5. Generally, the annotated training data may be based, in part, oncombinations of pixel intensities that are considered visual features.Initially, from a first dataset comprising a plurality of CBCT dentalimages, a manual segmentation tool may be applied in order to label the‘tooth’ pixels S530. In an embodiment, the manual segmentation tool canbe a ‘brush tool’. Following manual labeling of ‘tooth’ of each pixel ofeach dental image of the dataset, a pretrained convolutional neuralnetwork (CNN), referred to herein as “retrained CNN”, and CNN classifiertherein, may be trained to perform per pixel predictions of ‘tooth’based upon the labeled images of the first dataset. In an embodiment,the pretrained CNN classifier can be based on AlexNet. In anotherembodiment, the pretrained CNN classifier can be further tuned accordingto a plurality of labeled pixel patches S532. To this end, thepixel-wise manually segmented dental images described above areconverted to a plurality of pixel patches S531, wherein each of theplurality of generated pixel patches may comprise 120 pixels surroundinga central pixel. Compared with pixel-wise training, patch-wise trainingdecreases training time without unnecessarily sacrificing resultingclassification accuracy.

Following training, the retrained CNN classifier may be applied to asecond dataset comprising a plurality of CBCT dental images. In order toprepare the resulting retrained CNN classifier predictions for activecontour modeling, or snakes segmentation S533, the retrained CNNclassifier predictions can be converted to segmentations S534. Thesegmented predictions may then be downsampled to obtain prepared imagesfor snakes segmentation S535, a framework in computer vision fordelineating an object outline from a 2D image. To this end, pixels ofthe dental images may first be thresholded according to Hounsfield units(HU), wherein HU values reflect the radiodensity of a biologicalstructure S536. Table 1 describes exemplary HU thresholding values,according to an exemplary embodiment of the present disclosure. Everysecond pixel may be taken into a sample dataset to generate a Gaussianmodel estimator for given biological structure types, or classes S537,thus providing a speed-map for snakes segmentation. The above-describedsnakes segmentation may be performed as a final segmentation of themodified output of the retrained CNN classifier. The resulting pluralityof CBCT dental images segmented via snakes segmentation form an initialFCN training database. The initial FCN training database can then beused to retrain a pretrained Unet-FCN S538, referred to herein as“retrained FCN”, and an FCN classifier therein. In an embodiment, falsepixels adjacent to two true pixels may have added weight.

TABLE 1 Radiodensity Assignments Label Tissue Type Radiodensity (HU) #0Clear <−990 HU #1 Teeth Segmented by CNN #2 Bone Not teeth > 650 HU #3Soft tissue 0 HU < not teeth < 650 HU #4 Liquids −800 HU < not teeth < 0HU

In order to evaluate the predictive value of the retrained FCNclassifier, and improve its future predictive power, the initial FCNtraining database can be continuously updated through a process of 3Dprediction enhancement S539. Broadly, the 3D prediction enhancementprocess follows a similar, run-time, process employed for unknown imagesduring implementation of the retrained FCN classifier. First,predictions of the initial FCN training database by the retrained FCNclassifier may be segmented. These segmentations may then be convertedto a 3D polygonal surface. This allows for enhancement of the 3D surfacemodel at a holistic level and with focus on the result, eliminating thelaborious task of enhancing individual slices of the 3D polygonalsurface. Once enhanced, the 3D polygonal surface model can then beconverted back into segmentation and, upon confirming the segmentationquality, returned to a subsequent FCN training database.

Specifically, 3D prediction enhancement comprises surface reconstructionvia marching cubes, for instance, followed by manual adjustments toapply filters and correct prediction errors in the reconstructedsurface. With manual adjustments completed at the level of the 3D model,the surface may be re-segmented into 2D slices and returned to theinitial FCN training database, thus forming the subsequent trainingdatabase. When the subsequent FCN training database has doubled in size,the retrained FCN classifier may be further retrained on the enhanced,subsequent FCN training database S540 in order to further improveclassification accuracy. As described, the above process may beiterative.

According to an embodiment, the initial FCN training database andsubsequent FCN training databases, therefrom, comprise approximately50,000 dental images, based upon the quality of the produced data. Theenhancement and retraining process may be repeated when an FCN trainingdatabase doubles in size, the dental images with lowest predictionquality being enhanced, as described above, and the FCN classifier beingretrained in order to adjust prediction quality.

Each of the steps of the above-described training protocol will befurther discussed below.

FIG. 6A and FIG. 6B are illustrations of an aspect of amanually-segmented dental image of a tooth. According to an embodimentof the present disclosure, a source image 601 from a first datasetcomprising the plurality of CBCT dental images, shown in FIG. 6A, may bemanually segmented. Through manual segmentation, the user is able tomanually assign labels to each pixel, a process creating ground truthdata for training semantic segmentation protocols. In FIG. 6B, a manualsegmentation via ‘brush tool’ S630, for instance, allows for exactidentification and assignment of a ‘tooth’ label to appropriate pixelsof the source image 601. In another embodiment, the manual segmentationtool may be selected from a group including but not limited to floodfill tool, smart polygon tool, and polygon tool.

Following manual labeling of ‘tooth’ of each pixel of each dental imageof the first dataset, the first dataset may be used to train apretrained CNN classifier, such as, for instance, AlexNet, to performper pixel predictions of ‘tooth’. In an embodiment, the pretrained CNNclassifier may be further tuned according to a plurality of labeledpixel patches. To this end, the manually segmented dental imagesdescribed above may be converted to a plurality of pixel patches S531,wherein each of the plurality of generated pixel patches comprises 120pixels surrounding a central pixel. Using pixel patches instead ofindividual pixels decreases training time without unnecessarilysacrificing classification accuracy. FIG. 7A through FIG. 7U areillustrations of exemplary pixel patches of a dental image processingprotocol, according to an embodiment of the present disclosure. Thepretrained CNN classifier may be retrained on pixel patches from the‘tooth’ class S731, as illustrated in FIG. 7A through FIG. 7U andwherein [1 0] indicates ‘tooth’ and [0 1] indicates ‘not tooth’, inorder to predict whether the central pixel of each pixel patch belongsto the ‘tooth’ class.

Following training, the retrained CNN classifier may be applied to asecond dataset comprising a plurality of CBCT dental images. FIG. 8A,FIG. 8B, FIG. 8C, and FIG. 8D are illustrations of aspects of a snakessegmentation of a dental image processing protocol, according to anembodiment of the present disclosure. In order to prepare the retrainedCNN classifier predictions for snakes segmentation S833, thepredictions, generated for a source image 801, for instance, shown inFIG. 8A, may first be converted to segmentations. The segmentedpredictions S834, shown in FIG. 8B, may then be downsampled to obtainimages prepared for snakes segmentation S835. To this end, pixels of thedental images may then be thresholded according to HU S836, shown inFIG. 8C, wherein HU values reflect the radiodensity of a tissue andsimilar hues indicate similar tissue types. A Gaussian model estimatormay provide a speed map for snakes segmentation. The above-describedsnakes segmentation may then be performed as a final segmentation of themodified output of the retrained CNN classifier S833, as shown in FIG.8D.

The resulting plurality of CBCT dental images segmented via snakessegmentation form an initial FCN training database. The initial FCNtraining database can then be used to retrain a pretrained Unet-FCN.

As described above, and with reference to FIG. 9, predictions from theretrained FCN classifier may be evaluated and improved via a 3D modelenhancement. FIG. 9 is a flowchart of an aspect of a training datageneration protocol, according to an embodiment of the presentdisclosure, wherein the predictions from the retrained FCN classifiermay be improved via enhancement of a 3D model. First, predictions of theinitial FCN training database from the retrained FCN classifier may besegmented S968. These segmentations may then be converted to a 3Dpolygonal surface S969. This allows for enhancement of the 3D surfacemodel S970 at a holistic level and with focus on the result, eliminatingthe need to enhance individual slices of the model. In an example, thisreduces computational burden from five-hundred 2D slices to one 3Dpolygonal surface. Having improved the 3D surface model by applyingfilters and correcting prediction errors, extraneous anatomical data maythen be removed from the 3D surface model S971 in order to isolateanatomical features of interest. Once ‘enhanced’, the 3D surface modelcan be reverted to segmentation S972. Segmentations may then beconfirmed for quality S973, relative to a pre-determined errorthreshold, and added to a subsequent FCN training database if ofsufficient quality.

Following enhancement of CBCT dental images of the subsequent FCNtraining database, as described above, and retraining of the retrainedFCN classifier, the run-time in FIG. 3 may be implemented with theretrained FCN classifier. FIG. 10A, FIG. 10B, and FIG. 10C areillustrations of a segmentation of teeth of a dental image afterprediction by the retrained FCN classifier. In FIG. 10A, an illustrationof a segmentation of an anterior aspect of a dental arch from a CBCTdental image, according to an embodiment of the present disclosure, atransverse plane segmentation 1062 with overlaid ‘tooth’ predictions isobserved. A full dental arch segmentation in a transverse plane 1063, inFIG. 10B, illustrates a segmented FCN classifier prediction across across-section of the dental arch, highlighting the crowns of each tooth.In a sagittal plane view of an aspect of a superior, or upper, and aninferior, or lower, dental arch 1064, in FIG. 10C, a complete view of across-section of an aspect of each dental arch is visible, includingcrowns and roots.

Conversely, FIG. 11A, FIG. 11B, and FIG. 11C are illustrations of asegmentation of bone of a dental image after prediction by the retrainedFCN classifier, according to an embodiment of the present disclosure.After applying the retrained FCN classifier to an unknown source image1101, for instance, shown in FIG. 11A, a sagittal view of the mouth of apatient, wherein the skull is on the left side of the image, a ‘bone’classification is predicted. Follow classification of ‘bone’, continuousregions of ‘bone’ may be identified and relative centers of mass may becompared to determine anatomic identity, in the context of the dentalimage plane. As shown in FIG. 11B, a contiguous region of classified‘bone’ proximate to the skull, or anatomically superior, is identifiedas the maxilla 1165, while an inferior contiguous region is identifiedas the mandible 1166, as shown in FIG. 11C.

A plurality of processed CBCT dental images generated via prediction bythe retrained FCN classifier, similar to those observed in FIG. 10Athrough FIG. 11C, can be segmented and reconstructed to render a complex3D model of dentition in the context of periodontal tissues, asreflected in FIG. 4. To this end, following rendering of thesegmentation of the above-described predictions via the retrained FCNclassifier, the resulting simple 3D model may be further enhanced toprovide additional information related to the periodontal tissueenvironment. From the simple 3D model generated via marching cubes, forinstance, a density measurement and distance measurement can beperformed. FIG. 12A and FIG. 12B are flowcharts of a determination of adistance measurement and a density measurement of a three-dimensionalmodel, respectively, according to an embodiment of the presentdisclosure. To this end, the density measurement 1254 and the distancemeasurement 1255 of the periodontal space may be computed from thesurface reconstruction of the segmented predictions of the retrained FCNclassifier S1253.

The distance measurement 1254 comprises locating and computing, for eachpoint on the surface reconstruction describing the surface of a boneS1220, a distance to a nearest point on the surface reconstructionsdescribing the root S1221. This distance, therefore, reflects a volumeof alveolar process wherein a tooth may move.

The density measurement comprises locating and computing, from eachpoint on a surface reconstruction describing the surface of a rootS1224, a metric of the density of the surrounding tissue S1225. Thismetric may be determined on the basis of mean voxel intensitiessurrounding a vertex-point coordinate of the surface reconstruction,reflecting the spatial arrangement of bone and the ability of a tooth tomove, therein.

Once calculated for each point within the surface reconstruction, orsimple 3D model, the density measurement S1226 and distance measurementS1222 may be mapped to the simple 3D model, rendering it a complex 3Dmodel. To allow for rapid visualization of the distance measurement,with reference again to FIG. 4, varying distances are denoted via heatmap, wherein regions of minimal thickness and maximal thickness are ofvarying hues.

Having generated a ‘color’ mapped complex 3D model of an initial toothposition of dentition of a patient, virtual setup and visualization ofintermediary tooth movements during realignment can be envisioned. In anembodiment, a dental professional can prescribe a final tooth positionfor each tooth of each dental arch. Further, this final tooth positioncan be determined in context of a control arrangement, or dentition,shown in FIG. 13A and FIG. 13B. FIG. 13A and FIG. 13B are illustrationsof a three-dimensional model of dentition, according to an embodiment ofthe present disclosure. An inferior dental arch 1302, shown in FIG. 13A,and superior dental arch 1303, shown in FIG. 13B, represent an optimalcrown position. Using the dental control and optimal crown positions ofFIG. 13A and FIG. 13B, a tooth movement plan can be developed.

Specifically, as shown in FIG. 14A, FIG. 14B, and FIG. 14C,illustrations of a positioning of a three-dimensional model generatedfrom a processed dental image, intermediary stages of tooth movement canbe determined based upon a prescribed final tooth position and aninitial tooth position. The initial tooth position 1475 in FIG. 14Areflects a maligned dental arch and the varying thicknesses of alveolarprocess surrounding the root of the tooth. In an example, the roots of alateral incisor are deep to the buccal surface of the alveolar process,whereas an adjacent tooth, or a central incisor, may be relativelysuperficial with respect to the buccal surface of the alveolar process.In an embodiment, a prescribing dental professional determines that alateral incisor, indicated by the left arrow of the initial toothalignment 1475, need be rotated about a transverse axis, the roots ofthe lateral incisor being moved anteriorly and proximate to the buccalsurface of the alveolar process. At an intermediary stage 1476, shown inFIG. 14B, with the left arrow still indicating the lateral incisor, therequired movement has been initiated. The changing hue of the modelproximate the roots of the lateral incisor reflect this movement.Following subsequent intermediary movements, a final tooth position1477, shown in FIG. 14C, may be achieved. Consequently, the determinedthickness of the alveolar process between the root surface and thebuccal surface of the alveolar process is decreased, as indicated by theshifting hue at the left arrow of the complex 3D model of FIG. 14C.

According to an embodiment, intermediary tooth positions may bedetermined manually according to a final tooth position, an initialtooth position, and the movements of adjacent teeth. In anotherembodiment, intermediary tooth positions may be determinedautomatically, via a path determining protocol executed by theprocessing circuitry.

In another embodiment, determined tooth movements may be informed by aquantitative model of expected bone growth and resorption at the root,lingual, and buccal surfaces of the alveolar process. For example, as atooth movement results in anterior rotation of a root of a tooth, bonedeposition, and thus thickening, may occur on the buccal surface of thealveolar process. Concurrently, bone resorption may occur on the lingualsurface of the alveolar process. Expected bone growth or bone resorptioncan be added to the complex 3D model of the teeth and surrounding boneduring rendering of intermediary tooth movements.

In still another embodiment, upon evaluation of a complex 3D model, aprescribed final tooth position may not be a realizable final toothposition due to constraints of the facial skeleton, as informed by theabove-described density measurement and distance measurement. In suchcase, a realizable final tooth position is determined, with the input ofthe prescribing dental professional, and in the context of function andaesthetic.

Having determined intermediary and final tooth positions, and withreference again to FIG. 1, dental aligners may be fabricated,accordingly.

Next, a hardware description of the dental image processing deviceaccording to exemplary embodiments is described with reference to FIG.15. In FIG. 15, the dental image processing device includes a CPU 1580which performs the processes described above/below. In anotherembodiment, the processing device may be a GPU, GPGPU, or TPU. Theprocess data and instructions may be stored in memory 1581. Theseprocesses and instructions may also be stored on a storage medium disk1582 such as a hard drive (HDD) or portable storage medium or may bestored remotely. Further, the claimed advancements are not limited bythe form of the computer-readable media on which the instructions of theinventive process are stored. For example, the instructions may bestored on CDs, DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM,hard disk or any other information processing device with which thedental image processing device communicates, such as a server orcomputer.

Further, the claimed advancements may be provided as a utilityapplication, background daemon, or component of an operating system, orcombination thereof, executing in conjunction with CPU 1580 and anoperating system such as Microsoft Windows 7, UNIX, Solaris, LINUX,Apple MAC-OS and other systems known to those skilled in the art.

The hardware elements in order to achieve the dental image processingdevice may be realized by various circuitry elements, known to thoseskilled in the art. For example, CPU 1580 may be a Xenon or Coreprocessor from Intel of America or an Opteron processor from AMD ofAmerica, or may be other processor types that would be recognized by oneof ordinary skill in the art. Alternatively, the CPU 1580 may beimplemented on an FPGA, ASIC, PLD or using discrete logic circuits, asone of ordinary skill in the art would recognize. Further, CPU 1580 maybe implemented as multiple processors cooperatively working in parallelto perform the instructions of the inventive processes described above.

The dental image processing device in FIG. 15 also includes a networkcontroller 1583, such as an Intel Ethernet PRO network interface cardfrom Intel Corporation of America, for interfacing with network 1595. Ascan be appreciated, the network 1595 can be a public network, such asthe Internet, or a private network such as an LAN or WAN network, or anycombination thereof and can also include PSTN or ISDN sub-networks. Thenetwork 1595 can also be wired, such as an Ethernet network, or can bewireless such as a cellular network including EDGE, 3G and 4G wirelesscellular systems. The wireless network can also be WiFi, Bluetooth®, orany other wireless form of communication that is known.

The dental image processing device further includes a display controller1584, such as a NVIDIA GeForce GTX® or Quadro® graphics adaptor fromNVIDIA Corporation of America for interfacing with display 1585, such asa Hewlett Packard HPL2445w® LCD monitor. A general purpose I/O interface1586 interfaces with a keyboard and/or mouse 1587 as well as a touchscreen panel 1588 on or separate from display 1585. General purpose I/Ointerface also connects to a variety of peripherals 1589 includingprinters and scanners, such as an OfficeJet® or DeskJet® from HewlettPackard.

A sound controller 1590 is also provided in the dental image processingdevice, such as Sound Blaster X-Fi Titanium from Creative, to interfacewith speakers/microphone 1591 thereby providing sounds and/or music.

The general purpose storage controller 1592 connects the storage mediumdisk 1582 with communication bus 1593, which may be an ISA, EISA, VESA,PCI, or similar, for interconnecting all of the components of the dentalimage processing device. A description of the general features andfunctionality of the display 1585, keyboard and/or mouse 1587, as wellas the display controller 1584, storage controller 1592, networkcontroller 1583, sound controller 1590, and general purpose I/Ointerface 1586 is omitted herein for brevity as these features areknown.

Embodiments of the present disclosure may also be as set forth in thefollowing parentheticals.

(1) A method for generating an intermediate position of one or moreteeth of a dental arch of a patient, comprising classifying, viaprocessing circuitry, pixels of one or more medical images into classescorresponding to biological structure types, segmenting, via theprocessing circuitry, the classified pixels of the one or more medicalimages into biological structures, rendering, via the processingcircuitry, a three-dimensional model of the biological structures basedon the segmented classified pixels, determining, via the processingcircuitry, one or more metrics, based upon the three-dimensional model,describing characteristics of a bone of the biological structures,acquiring, via the processing circuitry, a final position of each of theone or more teeth of the dental arch based upon the three-dimensionalmodel, and generating, via the processing circuitry, the intermediateposition of each of the one or more teeth of the patient based upon theone or more metrics and the acquired final position.

(2) The method according to (1), wherein the classifying furthercomprises training, via the processing circuitry, a classifier on atraining database, and classifying, via the processing circuitry, thepixels of the one or more medical images based upon the classifier,wherein the training database includes a corpus of reference medicalimages, each reference medical image comprising at least oneidentifiable reference biological structure associated in the trainingdatabase with at least one corresponding description of the biologicalstructure type.

(3) The method according to either (1) or (2), wherein the intermediateposition of each of the one or more teeth is determined based upon aposition of an aspect of a proximate tooth of the one or more teeth ofthe dental arch.

(4) The method according to any of (1) to (3), wherein the trainingfurther comprises training, via the processing circuitry, a first neuralnetwork according to a first dataset, training, via the processingcircuitry, a second neural network according to a second dataset, thesecond dataset comprising a plurality of classification predictions ofthe first neural network, and generating, via the processing circuitry,the training database based upon a plurality of classificationpredictions of the second neural network.

(5) The method according to any of (1) to (4), wherein the second neuralnetwork is a fully convolutional neural network.

(6) The method according to any of (1) to (5), wherein one of the one ormore metrics is a distance metric, the distance metric being defined asa distance between a surface of a root of one of the one or more teethof the dental arch and a surface of an alveolar process.

(7) The method according to any of (1) to (6), wherein one of the one ormore metrics is a density metric, the density metric being defined as ameasure of mean intensity of voxels adjacent to a central voxel.

(8) A non-transitory computer-readable storage medium storingcomputer-readable instructions that, when executed by a computer havinga processing circuitry, cause the computer to perform a method, themethod comprising classifying pixels of one or more medical images intoclasses corresponding to biological structure types, segmenting theclassified pixels of the one or more medical images into biologicalstructures, rendering a three-dimensional model of the biologicalstructures based on the segmented classified pixels, determining one ormore metrics, based upon the three-dimensional model, describingcharacteristics of a bone of the biological structures, acquiring afinal position of each of the one or more teeth of the dental arch basedupon the three-dimensional model, and generating the intermediateposition of each of the one or more teeth of the patient based upon theone or more metrics and the acquired final position.

(9) The method according to (8), wherein the classifying furthercomprises training a classifier on a training database, and classifyingthe pixels of the one or more medical images based upon the classifier,wherein the training database includes a corpus of reference medicalimages, each reference medical image comprising at least oneidentifiable reference biological structure associated in the trainingdatabase with at least one corresponding description of the biologicalstructure type.

(10) The method according to either (8) or (9), wherein the intermediateposition of each of the one or more teeth is determined based upon aposition of an aspect of a proximate tooth of the one or more teeth ofthe dental arch.

(11) The method according to any of (8) to (10), wherein the trainingfurther comprises training a first neural network according to a firstdataset, training a second neural network according to a second dataset,the second dataset comprising a plurality of classification predictionsof the first neural network, and generating the training database basedupon a plurality of classification predictions of the second neuralnetwork.

(12) The method according to any of (8) to (11), wherein the secondneural network is a fully convolutional neural network.

(13) The method according to any of (8) to (12), wherein one of the oneor more metrics is a distance metric, the distance metric being definedas a distance between a surface of a root of one of the one or moreteeth of the dental arch and a surface of an alveolar process.

(14) The method according to any of (8) to (13), wherein one of the oneor more metrics is a density metric, the density metric being defined asa measure of mean intensity of voxels adjacent to a central voxel.

(15) An apparatus for processing of dental images, comprising aprocessing circuitry configured to classify pixels of one or moremedical images into classes corresponding to biological structure types,segment the classified pixels of the one or more medical images intobiological structures, render a three-dimensional model of thebiological structures based on the segmented classified pixels,determine one or more metrics, based upon the three-dimensional model,describing a bone of the biological structures, acquire a final positionof each of the one or more teeth of the dental arch based upon thethree-dimensional model, and generate the intermediate position of eachof the one or more teeth of the patient based upon the one or moremetrics and the acquired final position.

(16) The apparatus according to (15), wherein the processing circuitryis further configured to train a classifier on a training database, andclassify the pixels of the one or more medical images based upon theclassifier, wherein the training database includes a corpus of referencemedical images, each reference medical image comprising at least oneidentifiable reference biological structure associated in the trainingdatabase with at least one corresponding description of the biologicalstructure type.

(17) The apparatus according to either (15) or (16), wherein theintermediate position of each of the one or more teeth is determinedbased upon a position of an aspect of a proximate tooth of the one ormore teeth of the dental arch.

(18) The apparatus according to any of (15) to (17), wherein thetraining further comprises training a first neural network according toa first dataset, training a second neural network according to a seconddataset, the second dataset comprising a plurality of classificationpredictions of the first neural network, and generating the trainingdatabase based upon a plurality of classification predictions of thesecond neural network.

(19) The apparatus according to any of (15) to (18), wherein one of theone or more metrics is a distance metric, the distance metric beingdefined as a distance between a surface of a root of one of the one ormore teeth of the dental arch and a surface of an alveolar process.

(20) The apparatus according to any of (15) to (19), wherein one of theone or more metrics is a density metric, the density metric beingdefined as a measure of mean intensity of voxels adjacent to a centralvoxel.

(21) A method of pixel-based classification of medical images,comprising training a neural network to perform the pixel-basedclassification of the medical images, the training comprising performinga first training on a classifier using a first training set of medicalimages, pixels of the first training set of medical images beingmanually labeled, applying the classifier after the first training to asecond training set of medical images to classify pixels of the secondtraining set of medical images, identifying and correcting incorrectlyclassified pixels of the second training set of medical images, andperforming a second training on the classifier after the first trainingusing the manually labeled first training set of medical images and thecorrectly classified second training set of medical images, and applyingthe classifier after the second training to the medical images toperform the pixel-based classification, the pixel-based classificationof the medical images including assigning pixels of each medical imageto a biological structure type.

(22) The method of (21), wherein the training the neural network furthercomprises generating a third training set of medical images thatincludes the manually labeled first training set of medical images andthe correctly classified second training set of medical images,allocating each medical image of the third training set of medicalimages to one of a first subset of the third training set of medicalimages or a second subset of the third training set of medical images,and performing a third training on the classifier after the secondtraining using the first subset of the third training set of medicalimages.

(23) The method of either (21) or (22), wherein the training the neuralnetwork further comprises applying the classifier after the thirdtraining to the second subset of the third training set of medicalimages, identifying images of the second subset of the third trainingset of medical images that are incorrectly classified by the classifier,the identified images having a pixel classification error rate above apixel classification threshold, reallocating a number of the identifiedincorrectly classified images of the second subset of the third trainingset of medical images to the first subset of the third training set ofmedical images, and reallocating a number of images, corresponding tothe number of images reallocated to the first subset of the thirdtraining set of medical images, from the first subset of the thirdtraining set of medical images to the second subset of the thirdtraining set of medical images, and training the classifier after thethird training using the first subset of the third training set ofmedical images including the reallocated images of the second subset ofthe third training set of medical images.

(24) The method of any one of (21) to (23), wherein a cycle includingthe applying, the identifying, the reallocating, and the training isperformed two or more times.

(25) The method of any one of (21) to (24), wherein the training theneural network further comprises applying the classifier after the thirdtraining to the second subset of the third training set of medicalimages, identifying images of the second subset of the third trainingset of medical images that are incorrectly classified, the identifiedimages having a pixel classification error rate above a pixelclassification threshold, reallocating images between the first subsetof the third training set of medical images and the second subset of thethird training set of medical images based on a comparison of a quantityof the identified incorrectly classified images and an identificationthreshold, and training the classifier after the third training usingthe first subset of the third training set of medical images includingthe reallocated images the third training set of medical images.

(26) The method of any one of (21) to (25), wherein a cycle includingthe applying, the identifying, the reallocating, and the training isperformed two or more times.

(27) The method of any one of (21) to (26), wherein the neural networkis a fully convolutional neural network.

(28) The method of any one of (21) to (27), wherein the biologicalstructure type is one of a hard tissue or a soft tissue.

(29) An apparatus for pixel-based classification of medical images,comprising processing circuitry configured to train a neural network toperform the pixel-based classification of the medical images, thetraining comprising performing a first training on a classifier using afirst training set of medical images, pixels of the first training setof medical images being manually labeled, applying the classifier afterthe first training to a second training set of medical images toclassify pixels of the second training set of medical images,identifying and correcting incorrectly classified pixels of the secondtraining set of medical images, and performing a second training on theclassifier after the first training using the manually labeled firsttraining set of medical images and the correctly classified secondtraining set of medical images, and apply the classifier after thesecond training to the medical images to perform the pixel-basedclassification, the pixel-based classification of the medical imagesincluding assigning pixels of each medical image to a biologicalstructure type.

(30) The apparatus of (29), wherein the processing circuitry is furtherconfigured to train the neural network by generating a third trainingset of medical images that includes the manually labeled first trainingset of medical images and the correctly classified second training setof medical images, allocating each medical image of the third trainingset of medical images to one of a first subset of the third training setof medical images or a second subset of the third training set ofmedical images, and performing a third training on the classifier afterthe second training using the first subset of the third training set ofmedical images.

(31) The apparatus of either of (29) or (30), wherein the processingcircuitry is further configured to train the neural network by applyingthe classifier after the third training to the second subset of thethird training set of medical images, identifying images of the secondsubset of the third training set of medical images that are incorrectlyclassified by the classifier, the identified images having a pixelclassification error rate above a pixel classification threshold,reallocating a number of the identified incorrectly classified images ofthe second subset of the third training set of medical images to thefirst subset of the third training set of medical images, andreallocating a number of images, corresponding to the number of imagesreallocated to the first subset of the third training set of medicalimages, from the first subset of the third training set of medicalimages to the second subset of the third training set of medical images,and training the classifier after the third training using the firstsubset of the third training set of medical images including thereallocated images of the second subset of the third training set ofmedical images.

(32) The apparatus of any one of (29) to (31), wherein a cycle includingthe applying, the identifying, the reallocating, and the training isperformed two or more times.

(33) The apparatus of any one of (29) to (32), wherein the processingcircuitry is further configured to train the neural network by applyingthe classifier after the third training to the second subset of thethird training set of medical images, identifying images of the secondsubset of the third training set of medical images that are incorrectlyclassified, the identified images having a pixel classification errorrate above a pixel classification threshold, reallocating images betweenthe first subset of the third training set of medical images and thesecond subset of the third training set of medical images based on acomparison of a quantity of the identified incorrectly classified imagesand an identification threshold, and training the classifier after thethird training using the first subset of the third training set ofmedical images including the reallocated images the third training setof medical images.

(34) The apparatus of any one of (29) to (33), wherein a cycle includingthe applying, the identifying, the reallocating, and the training isperformed two or more times.

(35) A non-transitory computer-readable storage medium storingcomputer-readable instructions that, when executed by a computer, causethe computer to perform a method of pixel-based classification ofmedical images, the method comprising training a neural network toperform the pixel-based classification of the medical images, thetraining comprising performing a first training on a classifier using afirst training set of medical images, pixels of the first training setof medical images being manually labeled, applying the classifier afterthe first training to a second training set of medical images toclassify pixels of the second training set of medical images,identifying and correcting incorrectly classified pixels of the secondtraining set of medical images, and performing a second training on theclassifier after the first training using the manually labeled firsttraining set of medical images and the correctly classified secondtraining set of medical images; and applying the classifier after thesecond training to the medical images to perform the pixel-basedclassification, the pixel-based classification of the medical imagesincluding assigning pixels of each medical image to a biologicalstructure type.

(36) The non-transitory computer-readable storage medium of (35),wherein the training the neural network further comprises generating athird training set of medical images that includes the manually labeledfirst training set of medical images and the correctly classified secondtraining set of medical images, allocating each medical image of thethird training set of medical images to one of a first subset of thethird training set of medical images or a second subset of the thirdtraining set of medical images, and performing a third training on theclassifier after the second training using the first subset of the thirdtraining set of medical images.

(37) The non-transitory computer-readable storage medium of either (35)or (36), wherein the training the neural network further comprisesapplying the classifier after the third training to the second subset ofthe third training set of medical images, identifying images of thesecond subset of the third training set of medical images that areincorrectly classified by the classifier, the identified images having apixel classification error rate above a pixel classification threshold,reallocating a number of the identified incorrectly classified images ofthe second subset of the third training set of medical images to thefirst subset of the third training set of medical images, andreallocating a number of images, corresponding to the number of imagesreallocated to the first subset of the third training set of medicalimages, from the first subset of the third training set of medicalimages to the second subset of the third training set of medical images,and training the classifier after the third training using the firstsubset of the third training set of medical images including thereallocated images of the second subset of the third training set ofmedical images.

(38) The non-transitory computer-readable storage medium of any one of(35) to (37), wherein a cycle including the applying, the identifying,the reallocating, and the training is performed two or more times.

(39) The non-transitory computer-readable storage medium of any one of(35) to (38), wherein the training the neural network further comprisesapplying the classifier after the third training to the second subset ofthe third training set of medical images, identifying images of thesecond subset of the third training set of medical images that areincorrectly classified, the identified images having a pixelclassification error rate above a pixel classification threshold,reallocating images between the first subset of the third training setof medical images and the second subset of the third training set ofmedical images based on a comparison of a quantity of the identifiedincorrectly classified images and an identification threshold, andtraining the classifier after the third training using the first subsetof the third training set of medical images including the reallocatedimages the third training set of medical images.

(40) The non-transitory computer-readable storage medium of any one of(35) to (39), wherein a cycle including the applying, the identifying,the reallocating, and the training is performed two or more times.

Obviously, numerous modifications and variations are possible in lightof the above teachings. It is therefore to be understood that within thescope of the appended claims, the invention may be practiced otherwisethan as specifically described herein.

Thus, the foregoing discussion discloses and describes merely exemplaryembodiments of the present invention. As will be understood by thoseskilled in the art, the present invention may be embodied in otherspecific forms without departing from the spirit or essentialcharacteristics thereof. Numerous modification and variations on thepresent invention are possible in light of the above teachings.Accordingly, the disclosure of the present invention is intended to beillustrative, but not limiting of the scope of the invention, as well asother claims. The disclosure, including any readily discernible variantsof the teachings herein, defines, in part, the scope of the foregoingclaim terminology such that no inventive subject matter is dedicated tothe public.

All publications, patent applications, patents, and other referencesmentioned herein are incorporated by reference in their entirety.Further, the materials, methods, and examples are illustrative only andare not intended to be limiting, unless otherwise specified.

1. A method of pixel-based classification of medical images, comprising:training a neural network to perform the pixel-based classification ofthe medical images, the training comprising performing a first trainingon a classifier using a first training set of medical images, pixels ofthe first training set of medical images being manually labeled,applying the classifier after the first training to a second trainingset of medical images to classify pixels of the second training set ofmedical images, identifying and correcting incorrectly classified pixelsof the second training set of medical images, and performing a secondtraining on the classifier after the first training using the manuallylabeled first training set of medical images and the correctlyclassified second training set of medical images; and applying theclassifier after the second training to the medical images to performthe pixel-based classification, the pixel-based classification of themedical images including assigning pixels of each medical image to abiological structure type.
 2. The method of claim 1, wherein thetraining the neural network further comprises generating a thirdtraining set of medical images that includes the manually labeled firsttraining set of medical images and the correctly classified secondtraining set of medical images, allocating each medical image of thethird training set of medical images to one of a first subset of thethird training set of medical images or a second subset of the thirdtraining set of medical images, and performing a third training on theclassifier after the second training using the first subset of the thirdtraining set of medical images.
 3. The method of claim 2, wherein thetraining the neural network further comprises applying the classifierafter the third training to the second subset of the third training setof medical images, identifying images of the second subset of the thirdtraining set of medical images that are incorrectly classified by theclassifier, the identified images having a pixel classification errorrate above a pixel classification threshold, reallocating a number ofthe identified incorrectly classified images of the second subset of thethird training set of medical images to the first subset of the thirdtraining set of medical images, and reallocating a number of images,corresponding to the number of images reallocated to the first subset ofthe third training set of medical images, from the first subset of thethird training set of medical images to the second subset of the thirdtraining set of medical images, and training the classifier after thethird training using the first subset of the third training set ofmedical images including the reallocated images of the second subset ofthe third training set of medical images.
 4. The method of claim 3,wherein a cycle including the applying, the identifying, thereallocating, and the training is performed two or more times.
 5. Themethod of claim 2, wherein the training the neural network furthercomprises applying the classifier after the third training to the secondsubset of the third training set of medical images, identifying imagesof the second subset of the third training set of medical images thatare incorrectly classified, the identified images having a pixelclassification error rate above a pixel classification threshold,reallocating images between the first subset of the third training setof medical images and the second subset of the third training set ofmedical images based on a comparison of a quantity of the identifiedincorrectly classified images and an identification threshold, andtraining the classifier after the third training using the first subsetof the third training set of medical images including the reallocatedimages the third training set of medical images.
 6. The method of claim5, wherein a cycle including the applying, the identifying, thereallocating, and the training is performed two or more times.
 7. Themethod of claim 1, wherein the neural network is a fully convolutionalneural network.
 8. The method of claim I , wherein the biologicalstructure type is one of a hard tissue or a soft tissue.
 9. An apparatusfor pixel-based classification of medical images, comprising: processingcircuitry configured to train a neural network to perform thepixel-based classification of the medical images, the trainingcomprising performing a first training on a classifier using a firsttraining set of medical images, pixels of the first training set ofmedical images being manually labeled, applying the classifier after thefirst training to a second training set of medical images to classifypixels of the second training set of medical images, identifying andcorrecting incorrectly classified pixels of the second training set ofmedical images, and performing a second training on the classifier afterthe first training using the manually labeled first training set ofmedical images and the correctly classified second training set ofmedical images, and apply the classifier after the second training tothe medical images to perform the pixel-based classification, thepixel-based classification of the medical images including assigningpixels of each medical image to a biological structure type.
 10. Theapparatus of claim 9, wherein the processing circuitry is furtherconfigured to train the neural network by generating a third trainingset of medical images that includes the manually labeled first trainingset of medical images and the correctly classified second training setof medical images, allocating each medical image of the third trainingset of medical images to one of a first subset of the third training setof medical images or a second subset of the third training set ofmedical images, and performing a third training on the classifier afterthe second training using the first subset of the third training set ofmedical images.
 11. The apparatus of claim 10, wherein the processingcircuitry is further configured to train the neural network by applyingthe classifier after the third training to the second subset of thethird training set of medical images, identifying images of the secondsubset of the third training set of medical images that are incorrectlyclassified by the classifier, the identified images having a pixelclassification error rate above a pixel classification threshold,reallocating a number of the identified incorrectly classified images ofthe second subset of the third training set of medical images to thefirst subset of the third training set of medical images, andreallocating a number of images, corresponding to the number of imagesreallocated to the first subset of the third training set of medicalimages, from the first subset of the third training set of medicalimages to the second subset of the third training set of medical images,and training the classifier after the third training using the firstsubset of the third training set of medical images including thereallocated images of the second subset of the third training set ofmedical images.
 12. The apparatus of claim 11, wherein a cycle includingthe applying, the identifying, the reallocating, and the training isperformed two or more times.
 13. The apparatus of claim 10, wherein theprocessing circuitry is further configured to train the neural networkby applying the classifier after the third training to the second subsetof the third training set of medical images, identifying images of thesecond subset of the third training set of medical images that areincorrectly classified, the identified images having a pixelclassification error rate above a pixel classification threshold,reallocating images between the first subset of the third training setof medical images and the second subset of the third training set ofmedical images based on a comparison of a quantity of the identifiedincorrectly classified images and an identification threshold, andtraining the classifier after the third training using the first subsetof the third training set of medical images including the reallocatedimages the third training set of medical images.
 14. The apparatus ofclaim 13, wherein a cycle including the applying, the identifying, thereallocating, and the training is performed two or more times.
 15. Anon-transitory computer-readable storage medium storingcomputer-readable instructions that, when executed by a computer, causethe computer to perform a method of pixel-based classification ofmedical images, the method comprising: training a neural network toperform the pixel-based classification of the medical images, thetraining comprising performing a first training on a classifier using afirst training set of medical images, pixels of the first training setof medical images being manually labeled, applying the classifier afterthe first training to a second training set of medical images toclassify pixels of the second training set of medical images,identifying and correcting incorrectly classified pixels of the secondtraining set of medical images, and performing a second training on theclassifier after the first training using the manually labeled firsttraining set of medical images and the correctly classified secondtraining set of medical images; and applying the classifier after thesecond training to the medical images to perform the pixel-basedclassification, the pixel-based classification of the medical imagesincluding assigning pixels of each medical image to a biologicalstructure type.
 16. The non-transitory computer-readable storage mediumof claim 15, wherein the training the neural network further comprisesgenerating a third training set of medical images that includes themanually labeled first training set of medical images and the correctlyclassified second training set of medical images, allocating eachmedical image of the third training set of medical images to one of afirst subset of the third training set of medical images or a secondsubset of the third training set of medical images, and performing athird training on the classifier after the second training using thefirst subset of the third training set of medical images.
 17. Thenon-transitory computer-readable storage medium of claim 16, wherein thetraining the neural network further comprises applying the classifierafter the third training to the second subset of the third training setof medical images, p1 identifying images of the second subset of thethird training set of medical images that are incorrectly classified bythe classifier, the identified images having a pixel classificationerror rate above a pixel classification threshold, reallocating a numberof the identified incorrectly classified images of the second subset ofthe third training set of medical images to the first subset of thethird training set of medical images, and reallocating a number ofimages, corresponding to the number of images reallocated to the firstsubset of the third training set of medical images, from the firstsubset of the third training set of medical images to the second subsetof the third training set of medical images, and training the classifierafter the third training using the first subset of the third trainingset of medical images including the reallocated images of the secondsubset of the third training set of medical images.
 18. Thenon-transitory computer-readable storage medium of claim 17, wherein acycle including the applying, the identifying, the reallocating, and thetraining is performed two or more times.
 19. The non-transitorycomputer-readable storage medium of claim 16, wherein the training theneural network further comprises applying the classifier after the thirdtraining to the second subset of the third training set of medicalimages, identifying images of the second subset of the third trainingset of medical images that are incorrectly classified, the identifiedimages having a pixel classification error rate above a pixelclassification threshold, reallocating images between the first subsetof the third training set of medical images and the second subset of thethird training set of medical images based on a comparison of a quantityof the identified incorrectly classified images and an identificationthreshold, and training the classifier after the third training usingthe first subset of the third training set of medical images includingthe reallocated images the third training set of medical images.
 20. Thenon-transitory computer-readable storage medium of claim 19, wherein acycle including the applying, the identifying, the reallocating, and thetraining is performed two or more times.